Fast substring fulltext search

ABSTRACT

A system includes receiving a substring to be searched and determining one or more N-grams associated with the received substring. A lookup of each of the one or more N-grams is performed using an N-gram index. A set of tokens associated with each of the one or more N-grams is calculated and a determination is made if the substring exits in the calculated set of tokens. The results of the determination if the substring exits in the set of tokens are outputted.

BACKGROUND

A full-text search comprises a technique for searching one or morecomputer-stored documents or a collection of documents that are storedin a database. A full-text search may differ from searches based onmetadata or searches based solely on parts of the texts (e.g., titles,abstracts, selected sections, or bibliographical references).

In a conventional full-text search, a search engine may examine eachword in every stored document as the full-text search attempts to matchsearch criteria such as a specific term or a phrase that is specified bya user (e.g., a text string). Full-text search applications may have tohandle large sets of data. For example, there may be millions ofdocuments that need to be searched for a single text string.

When using a conventional full-text search to search for a term orphrase by scanning each document for the term or phrase, a searchruntime, and corresponding CPU consumption, may be extremely high.Therefore, a full-text search that can reduce a search runtime and CPUconsumption is desirable.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a method according to some embodiments.

FIG. 2 illustrates a flow diagram according to some embodiments.

FIG. 3 is a flow diagram according to some embodiments.

FIG. 4 illustrates data strings according to some embodiments.

FIG. 5 illustrates a token index according to some embodiments.

FIG. 6 illustrates an N-gram index according to some embodiments.

FIG. 7 is a block diagram of a hardware system according to someembodiments.

DETAILED DESCRIPTION

The following description is provided to enable any person in the art tomake and use the described embodiments. Various modifications, however,will remain readily apparent to those in the art.

The following embodiments relate to a novel full-text search associatedwith data strings of text, or other data, that, instead of scanning alldata strings for the search term. Furthermore, the following embodimentsimplement a full-text search by tokenizing or segmenting the data to besearched, creating a token index (e.g., an inverted token index) andaccessing the token index while searching for a particular term orphrase to achieve faster response times. The present embodiments mayreduce CPU consumption during full-text searches.

Now referring to FIG. 1, a method 100 is illustrated according to someembodiments. At 110, a received data string is tokenized and a tokenindex is created. Referring now to FIG. 2, an embodiment of a process200 that relates to tokenizing a data string and creating a token indexassociated with the data string is illustrated. At 210, a received datastring is tokenized. Tokenizing may comprise splitting up the datastring into individual portions (i.e., tokens). For example, someapproaches to tokenizing a data string may comprise splitting the dataat a pre-defined set of characters such as, but not limited to, spaces,tabs, commas or dashes.

For illustrative purposes, and to aid in understanding features of thespecification, an example will now be introduced. This example is notintended to limit the scope of the claims. For example, and nowreferring to FIG. 4, the following set of four data strings may betokenized. Data string zero (“S0”) comprises “lipopoly saccharide”, datastring one (“S1”) comprises “neopolyol ester”, data string two (“S2”)comprises “cyclopolyenyl metal complex” and data string three (“S3”)comprises “lyposol metal ag”.

Tokenizing S0, based on spaces, may produce two tokens: “lipopoly” and“saccharide”. Tokenizing S1, based on spaces, may produce two tokens:“neopolyol” and “ester”. Tokenizing S2, based on spaces, may producethree tokens: “cyclopolyenyl”, “metal” and “complex”. Tokenizing S3,based on spaces, may produce three tokens: “lyposol”, “metal” and “ag”.In the present example, both S2 and S3 produced a same token of “metal”.

Each token may be added to a token index such as the token index 500 ofFIG. Continuing with the above example, and as illustrated in FIG. 5, asa result of tokenizing S0 is broken down into two tokens, T5 and T9; S1is broken down into two tokens, T4 and T8; S2 is broken down into threetokens, T2, T3 and T7; and S3 is broken down into three tokens, T1, T6and T7. As can be seen from FIG. 5, the token T7, metal, is found inboth S2 and S3.

In some embodiments, the token index may comprise an inverted tokenindex that stores (i) a token identifier and (ii) an identifier of thedata string that the token was derived from. An inverted index maycomprise an index data structure that stores a mapping from content,such as words or numbers, to its locations in a database file, or in adocument or a set of documents.

Referring back to FIG. 2, at 220 the token index may be updated (e.g.,populated) with token information. Next, at 230, a determination is madeif all tokens for a particular string have been processed. If all tokenshave been processed, the process of tokenizing and indexing a particulardata string ends at 240 else the process loops to continue updating thetoken index.

Referring back to FIG. 1, at 120 a substring to be searched is receivedat a full-text search engine. The substring may comprise a plurality ofASCII characters or symbols. Continuing with the above example, a firstsubstring of “poly” may be received at a full-text search engine. In asecond example, a second substring of “metal” may be received at thefull-text search engine. Both “poly” and “metal” may comprise substringsto be searched. Each substring may first be tokenized. Tokenizing thesubstring may be accomplished in exactly the same way as in the datastring was tokenized as described above. Each search token can beprocessed separately. In the present example, each search stringcomprises a single token. However, a search string may comprise multipletokens.

Next, at 130, one or more N-grams associated with the received substringare determined. An N-gram may comprise a sequence of N adjacent elementsin a string such as letters, numbers or symbols where N is an integer.For example, when N=2, a 2-gram is referred to as a bigram and when N=3,a 3-gram may be referred to as a trigram. In the present examples, andbased on using bigrams, “poly” may be broken up using bigrams such as“po”, “ol” and “ly” and “metal” may be broken up using bigrams such as“me”, “et”, “ta”, and “al”.

A lookup (e.g., a database query) may be performed against an N-gramindex for each of the one or more N-grams at 140. An N-gram index maycomprise an index of each of the tokens that were broken down intoN-grams along with an indication of which token or tokens each N-gram isassociated with. In some embodiments, the N-gram index may comprise aninverted index. For example, and referring to FIG. 6, an embodiment ofan N-gram index 600 is illustrated. N-gram index 600 may illustrate abigram index that indicates a plurality of bigrams 610 and theirassociated tokens 620. Continuing with the above examples, bigrams “po”,“ol” and “ly” may be looked up in the N-gram index (which in thisexample is a bigram inverted index). Likewise, “me”, “et”, “ta”, and“al” may also be looked up in the N-gram index.

Next, at 150, a set of tokens associated with each of the one or moreN-grams may be calculated. The calculation may comprise looking up(e.g., querying) each N-gram in the N-gram index and determining whichtoken(s) are associated with each N-gram. Continuing with the aboveexamples, bigrams “po”, “ol” and “ly” may be looked up in N-gram index600 and may yield a result of a set of token candidates. In thisexample, the set of token candidates comprises tokens [T3, T5, T6, T8].Similarly, “me”, “et”, “ta”, and “al” may be looked up in the N-gramindex 600 and may yield a result of a set of token candidates comprising[T7].

At 160 it is determined if the substring to be searched exits in thecalculated set of tokens. Since the set of token candidates may containfalse positives due to the N-gram index not containing information aboutan order of the N-grams in the token string, each token candidate (e.g.,T3, T5, T6, T8) may be examined to determine if the token candidateactually contains the substring to be searched (e.g., poly). In thepresent example, Token T6, lyposol, contains all three bigrams “po”,“ol”, and “ly” but the bigrams are not in a correct order (i.e., thebigrams are in an order of “ly”, “po”, and “ol” and the bigrams are notin an order of “po”, “ol”, and “ly”). Hence, T6 is removed from theresulting token list [T3, T5, T8]. In the second example, since thebigrams “me”, “et”, “ta”, and “al” are only found in a single token, T7,this step would result in a positive determination that the tokens arein a correct order. The correct order may be based on a type of languagebeing searched. For example, the English language is read from left toright while the Hebrew language is read from right to left.

At 170, results of the determination if the substring exits in thecalculated set of tokens are outputted. Outputting the results maycomprise looking up the data strings that are associated with the tokenscomprising the searched substring. This lookup may be performed againstthe token index 500. In the present example, token T3 is associated withS2, token T5 is associated with S0 and token T8 is associated with S1.Therefore, the output may comprise a set of data strings comprising [S0,S1, S2]. Similarly for “me”, “et”, “ta”, and “al”, token T7 isassociated with S2 and S3 so the results would be a set of data stringscomprising [S2, S3].

Now referring to FIG. 3, a flow diagram of a process 300 is illustratedaccording to some embodiments. At 305, a search string is tokenized. Thesearch string may be tokenized using a same tokenization algorithm aswas used during the population of a token index associated with one ormore data strings. For example, if a data string was tokenized (e.g.,split into portions) based on spaces, then a string to be searched mayalso be tokenized based on spaces. In some embodiments, a sophisticatedtoken separation algorithm may be utilized. For example, tokenseparators or more advanced token separation algorithms like adictionary lookup and out of vocabulary (“OOV”) word identification maybe used (e.g. for Asian languages) in lieu of spaces, commas, etc.

At 310, a determination is made if a search to be performed is (i) asubstring search such as described with respect to FIG. 1 or (ii) asimilar search which will be described below. The determination may bebased on an input by a user.

If a substring search is selected, one or more N-grams associated withthe tokenized substring are determined at 315. As described in the aboveexamples, if a search string of “poly” is input and bigrams areimplemented, then “poly” may be broken up using bigrams such as “po”,“ol” and “ly”.

Next, at 320, all N-grams are looked up in an N-gram index. Continuingwith the above example, each of the bigrams “po”, “ol” and “ly” may belooked up in a N-gram index (e.g., a bigram index for this example) suchas the N-gram index illustrated in FIG. 6. The results of looking up theN-grams in the N-gram index may be stored in a token list. For example,in the case of looking up “po”, “ol” and “ly”, and as illustrated inFIG. 6, “po” may be found in tokens [T3, T5, T6, T8], “ol” may be foundin tokens [T3, T5, T6, T8] and “ly” may be found in tokens [T3, T5, T6,T8]. Therefore, the token list comprises the following three sets oftoken identifications (“IDs”):

1. [T3, T5, T6, T8]

2. [T3, T5, T6, T8]

3. [T3, T5, T6, T8]

Next, at 325, an intersection of all token IDs from the token list iscreated. In the present example, intersecting the three sets of tokenIDs in the token list provides [T3, T5, T6, T8]∩[T3, T5, T6, T8]∩[T3,T5,T6, T8] which yields a result of [T3, T5, T6, T8]. In some embodiments,instead of using an intersection operation, a union of the elements mayalso be performed. In the present example, a union of the elements wouldproduce a same result.

At 330, a determination is made if the search substring exists in theintersection of the token list. The intersection may contain falsepositives because the N-gram index of FIG. 6 does not containinformation about an order of the bigrams in the data string so thetoken candidates from the intersection results [T3, T5, T6, T8] are tobe examined individually to determine if they actually contain thesubstring poly.

Continuing with the above example, token T3, cyclopolyenyl, contains thestring poly. Token T5, lipoply, contains the string poly. Token T8,neopolyol, contains the string poly. However, token T6, lyposol, containall three bigrams “po”, “ol”, and “ly” but they are not in the correctorder. As illustrated in FIG. 5, the order of the bigrams is “ly”, “po”and then “ol”. Therefore, T6 is removed from the token list which yieldsa result of [T3, T5, T8].

At 335, the resulting tokens are looked up in the token index todetermine a particular string(s) that each token is associated with. Forexample, T3=S2, T5=S0, T5=S1. Then, at 340, an intersection of allstring IDs are determined to produce a result set. For example, thefinal search result may comprise the union of the three strings based onthe inverted token index: [S2]∪[S0]∪[S1]=[S0, S1, S2]. At 345 adetermination is made if all tokens have been processed and if so, theprocess ends at 350.

Referring back to 310, if the determination is to use a similar search(e.g., not use a substring search), then the process 300 advances to 335and a token is looked up in the token index. Each search token may belooked up in the token index to determine a list of string identifiersfor each respective token. A similar search may comprise a fuzziersearch since it does not determine if a substring exits in anintersection or union of token IDs. For the similar search, theintersection of all string identifiers may be calculated to get the fullresult list.

This similar search may be useful if a slight difference between thesearch string and the indexed token exists. This may comprise the caseif the data quality is not perfect or the user entered a search wordwith a typo like “conplex” instead of “complex”.

The results of the similar search may be scored based on a ratio of theconcordant N-grams to a total number of N-grams of a token that arecalculated. If this ratio is above a selected threshold then a resultingtoken or tokens may be considered a match. For example, if a number ofconcordant bigrams is used as a measure for similarity, the bigrams for“complex” are [“co”, “om”, “mp”, “pl”, “le”, “ex”], and [“co”, “on”,“np”, “pl”, “le”, “ex”] for “conplex”. In this example 4 out of 6bigrams match. If a ratio of 4/6 is considered to be similar then thetokens associated with the results may be considered a match. The ratioto determine if tokens are similar may be a user or system determinedvariable. Based on the above-example, a token T2 may be considered amatch and token T2 may be looked up in the inverted token index withstring S2 as result.

In some embodiments related to multi-token searches, instead of a unionor intersection operation, different logic combinations may be appliedto the token list as was described at 330. For example, if multipletokens were searched, instead of a simple intersection of a token list,logic combinations may be applied to the tokens such as, for example,(e.g., a OR b AND NOT c). Furthermore, the present embodiments maysupport complex pattern searches like *c*poly* using single characterplaceholders. In some embodiments, a pattern search and a similar searchmay be combined. For example, a pattern search and a similar search maybe run consecutively or jointly on a search string.

In some embodiments, the tokenization of search strings may comprisestemming where stemming comprises a heuristic process that removesportions of the search string. For example, stemming a search string mayremove an end of a word being tokenized so that a root of the word istokenized.

Now referring to FIG. 7, an embodiment of an apparatus 700 isillustrated. The apparatus 700 may be associated with a search engine.The apparatus 700 may comprise a storage device 701, a medium 702, aprocessor 703 and a main memory 704. According to some embodiments, theapparatus 700 may further comprise a digital display port, such as aport adapted to be coupled to a digital computer monitor, television,portable display screen, or the like.

The main memory 702 may comprise any type of memory for storing data,such as, but not limited to, a Secure Digital (SD) card, a micro SDcard, a Single Data Rate Random Access Memory (SDR-RAM), a Double DataRate Random Access Memory (DDR-RAM), or a Programmable Read Only Memory(PROM). The main memory 702 may comprise a plurality of memory modules.

The processor 703 may include or otherwise be associated with dedicatedregisters, stacks, queues, etc. that are used to execute program codeand/or one or more of these elements may be shared there between. Insome embodiments, the processor 703 may comprise an integrated circuit.In some embodiments, the processor 703 may comprise circuitry to performa method such as, but not limited to, the method described with respectto FIG. 1, FIG. 2 and FIG. 3.

The processor 703 communicates with the storage device 701. The storagedevice 701 may comprise any appropriate information storage device,including combinations of magnetic storage devices (e.g., a hard diskdrive), optical storage devices, and/or semiconductor memory devices.The storage device 701 stores a program for controlling the processor703. The processor 703 performs instructions of the program, and therebyoperates in accordance with any of the embodiments described herein. Forexample, the processor 703 may tokenize a data string and a searchstring.

The medium 702 may comprise any computer-readable medium that may storeprocessor-executable instructions to be executed by the processor 703.For example, the medium 702 may comprise a non-transitory tangiblemedium such as, but is not limited to, a compact disk, a digital videodisk, flash memory, optical storage, random access memory, read onlymemory, or magnetic media.

As used herein, information may be “received” by or “transmitted” to,for example: (i) the apparatus 700 from another device; or (ii) asoftware application or module within the apparatus 700 from anothersoftware application, module, or any other source.

In some embodiments, the storage device 701 stores a database (e.g.,including information associated with N-grams and algorithms.).Moreover, various databases might be split or combined in accordancewith any of the embodiments described herein.

Embodiments described herein are solely for the purpose of illustration.Those in the art will recognize other embodiments may be practiced withmodifications and alterations to that described above.

What is claimed is:
 1. A system comprising: a processor; and anon-transitory computer-readable medium storing processor-executableprocess steps executable by the processor to cause the system to:receive a substring to be searched at a full-text search engine, thefull-text search engine comprising a processor; determine, via theprocessor, one or more substring N-grams associated with the receivedsub string; perform a lookup of each of the one or more substringN-grams in an N-gram index, the N-gram index comprising, for eachindexed N-gram, one or more associated token identifiers, each of thetoken identifiers identifying a token which contains the associatedindexed N-gram; calculate a set of tokens associated with each of theone or more substring N-grams based on the performed lookup; determine,via the processor, if the substring exists in the calculated set oftokens by determining an order of the one or more substring N-gramswithin each token of the calculated set of tokens; and in a case thatthe order of the one or more substring N-grams within a first token ofthe calculated set of tokens is determined to be out of order, removingthe first token from the calculated set of tokens.
 2. A system accordingto claim 1, wherein the N-grams comprise bigrams.
 3. A system accordingto claim 1, wherein the N-grams comprise trigrams.
 4. A system accordingto claim 1, wherein the set of tokens associated with each of the one ormore N-grams comprises an intersection of tokens associated with each ofthe one or more N-grams.
 5. A system according to claim 1, wherein theset of tokens associated with each of the one or more N-grams comprisesa union of tokens associated with each of the one or more N-grams.
 6. Asystem according to claim 1, wherein the processor-executable processfurther cause the processor to: output results of the determination ifthe substring exists in the set of tokens.
 7. A method implemented by acomputing system in response to execution of program code by a processorof the computing system, the method comprising: receiving a substring tobe searched at a full-text search engine, the full-text search enginecomprising a processor; determining, via the processor, one or moresubstring bigrams associated with the received substring; performing alookup of each of the one or more substring bigrams in a bigrams index,the bigrams index comprising, for each indexed bigram, one or moreassociated token identifiers, each of the token identifiers identifyinga token which contains the associated indexed bigram; calculating a setof tokens associated with each of the one or more substring bigramsbased on the performed lookup; determining, via the processor, if thesubstring exists in the calculated set of tokens by determining an orderof the one or more substring bigrams within each token of the calculatedset of tokens; in a case that the order of the one or more sub stringbigrams within a first token of the calculated set of tokens isdetermined to be out of order, removing the first token from thecalculated set of tokens; and outputting results of the determination ifthe substring exists in the calculated of tokens.
 8. A method accordingto claim 7, wherein calculating a set of tokens associated with each ofthe one or more bigrams is based on a union operation.
 9. A methodaccording to claim 7, wherein calculating a set of tokens associatedwith each of the one or more bigrams is based on an intersectionoperation.
 10. A method according to claim 7, wherein the set of tokensassociated with each of the one or more N-grams comprises anintersection of tokens associated with each of the one or more bigrams.11. A method according to claim 7, wherein the set of tokens associatedwith each of the one or more N-grams comprises a union of tokensassociated with each of the one or more bigrams.
 12. A system method toclaim 7, further comprising: outputting results of the determination ifthe substring exists in the set of tokens.
 13. A non-transitorycomputer-readable medium storing program code, the program codeexecutable by a processor of a computing system to cause the computingsystem to: receiving a substring to be searched at a full-text searchengine, the full-text search engine comprising a processor; determining,via the processor, one or more substring N-grams associated with thereceived sub string; performing a lookup of each of the one or more substring N-grams in an N-gram index, the N-gram index comprising, for eachindexed N-gram, one or more associated token identifiers, each of thetoken identifiers identifying a token which contains the associatedindexed N-gram; calculating a set of tokens associated with each of theone or more substring N-grams based on the performed lookup, wherein theset of tokens associated with each of the one or more N-grams comprisesan intersection of tokens associated with each of the one or moreN-grams; determining, via the processor, if the substring exists in thecalculated set of tokens by determining an order of the one or moresubstring N-grams within each token of the calculated set of tokens; ina case that the order of the one or more substring N-grams within afirst token of the calculated set of tokens is determined to be out oforder, removing the first token from the calculated set of tokens; andoutputting results of the determination if the substring exists in thecalculated set of tokens.
 14. A medium according to claim 13, whereinthe N-grams comprise bigrams.
 15. A medium according to claim 13,wherein the N-grams comprise trigrams.
 16. A medium according to claim13, wherein the set of tokens associated with each of the one or moreN-grams comprises a union of tokens associated with each of the one ormore N-grams.
 17. A medium according to claim 13, further comprising:outputting results of the determination if the substring exists in theset of tokens.
 18. A medium according to claim 13, wherein the order ofthe N-grams within the token is based on a language being searched.